AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generation

Recently, medical report generation, which aims to automatically generate a long and coherent descriptive paragraph of given image, has received growing research interests. Different from the general image captioning tasks, generation is more challenging for data-driven neural models. This mainly due 1) serious data bias: normal visual regions dominate dataset over abnormal regions, 2) very sequence. To alleviate above two problems, we propose an AlignTransformer framework, includes Align Hierarchical Attention (AHA) Multi-Grained Transformer (MGT) modules: AHA module first predicts disease tags input then learns multi-grained features by hierarchically aligning tags. The acquired disease-grounded can better represent could bias problem; MGT effectively uses framework report. experiments on public IU-Xray MIMIC-CXR datasets show that achieve results competitive with state-of-the-art methods datasets. Moreover, human evaluation conducted professional radiologists further proves effectiveness our approach.